Digital camera with selectively increased dynamic range by control of parameters during image acquisition

ABSTRACT

A parameter of a digital camera or other digital image acquisition device is adjusted to maintain the resulting digital signal within a range carried by a digital processing path that carries a limited number of bits. The magnitude of the parameter is then also used to represent the image. Examples of the parameter include analog signal gain and exposure time. This is a cost effective way to increase the dynamic range of a digital camera, instead of increasing the width of its digital processing path. The digital data may be processed to obtain either tone mapped images, which are compatible with current cameras and other equipment, or to obtain images with a greater dynamic range for use with suitable displays and the like.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation-in-part of U.S. patent application Ser. No.11/467,993, filed by Shimon Pertsel and Ohad Meitav on Aug. 29, 2006 nowU.S. Pat. No. 7,714,903, entitled “Wide Dynamic Range Image CapturingSystem Method and Apparatus.”

BACKGROUND

This application relates to automatic control of digital cameras andother electronic digital image acquisition devices, and the use ofvisual data generated by them, in a manner that selectively increasesthe dynamic range of the cameras or other devices.

Electronic cameras image scenes onto a two-dimensional sensor such as acharge-coupled-device (CCD), a complementary metal-on-silicon (CMOS)device or other type of light sensor. These devices include a largenumber of photo-detectors (typically two, three, four or more million)arranged across a small two dimensional surface that individuallygenerate a signal proportional to the intensity of light or otheroptical radiation (including infrared and ultra-violet regions of thespectrum adjacent the visible light wavelengths) striking the element.These elements, forming pixels of an image, are typically scanned in araster pattern to generate an analog signal with a time varyingmagnitude representative of the intensity of radiation striking onesensor element after another as they are scanned. Color data are mostcommonly obtained by using photo-detectors that are sensitive to each ofdistinct color components (such as red, green and blue), alternatelydistributed across the sensor.

A popular form of such an electronic camera is a small hand-held digitalcamera that records data of a large number of picture frames either asstill photograph “snapshots” or as sequences of frames forming a movingpicture. A significant amount of image processing is typically performedon the data of each frame within the camera before storing on aremovable non-volatile memory such as a magnetic tape cartridge, a flashmemory card, a recordable optical disc or a removable hard disk drive.The processed data are typically displayed as a reduced resolution imageon a liquid crystal display (LCD) device on the outside of the camera.The processed data are also typically compressed before storage in thenon-volatile memory in order to reduce the amount of storage capacitythat is taken by the data for each picture frame.

The data acquired by the image sensor are typically processed tocompensate for imperfections of the camera and to generally improve thequality of the image obtainable from the data. The correction for anydefective pixel photodetector elements of the sensor is one processingfunction. Another is white balance correction wherein the relativemagnitudes of different pixels of the primary colors are set torepresent white. This processing may also include de-mosaicing theindividual pixel data, when obtained from a type of sensor havingspatially separate monochromatic pixel detectors, in order to rendersuperimposed multi-colored pixels in the image data. This de-mosaicingthen makes it desirable to process the data to enhance and smooth edgesof the image. Compensation of the image data for noise and variations ofthe camera optical system across the image and for variations among thesensor photodetectors is also typically performed within the camera.Other processing typically includes one or more of gamma correction,contrast stretching, chrominance filtering and the like.

Electronic cameras also nearly always include an automatic exposurecontrol capability that sets the exposure time, size of its apertureopening and analog electronic gain of the sensor to result in theluminescence of the image or succession of images being at a certainlevel based upon calibrations for the sensor being used and userpreferences. These exposure parameters are calculated in advance of thepicture being taken, and then used to control the camera duringacquisition of the image data. For a scene with a particular level ofillumination, a decrease in the exposure time is made up by increasingthe size of the aperture or the gain of the sensor, or both, in order toobtain the data within a certain luminescence range. An increasedaperture results in an image with a reduced depth of field and increasedoptical blur, and increasing the gain causes the noise within the imageto increase. Conversely, when the exposure time can be increased, suchas when the scene is brightly lighted, the aperture and/or gain arereduced, which results in the image having a greater depth of fieldand/or reduced noise. In addition to analog gain being adjusted, or inplace of it, the digital gain of an image is often adjusted after thedata have been captured.

Digital camera systems typically have a limited dynamic range. Whilefilm cameras typically provide images with intensity dynamic rangesranging from 70 dB to 80 dB, current digital cameras typically offerdynamic ranges of less than 60 dB. One cause of this is the use ofdigital processing paths within the cameras that are limited in thenumber of bits they can carry. The cost of a digital camera can increaserapidly as the width of digital processing paths is increased. As thenumber of bits of data that an analog-to-digital converter or digitalsignal processor can process increases, as an example, the cost of acamera can increase significantly. The dynamic range of digital camerasis therefore typically maintained lower than that which can be obtainedby film cameras in order to be cost competitive.

This often results in one or more regions of an image having a wideintensity dynamic range being saturated, either positively ornegatively, or both. Any details of the image in saturated regions arelost since these regions are uniformly bright or dark, respectively.

SUMMARY

It is therefore desirable to be able to increase the dynamic range ofdigital cameras and other electronic digital image acquisition deviceswithout increasing the widths of their digital processing paths. Thismay be accomplished by controlling a parameter that makes sure that theimage signal remains within the range of the digital processing pathwhen being captured, in order to avoid the effects of saturation. Thatis, if a digital processing path is 8, 10 or 12 bits wide, for example,the image data are captured within a range that may be represented bysuch a number of bits. The image is therefore represented by these dataplus values of the parameter during acquisition of the data.

In embodiments described hereinafter, the controlled parameter is theanalog signal gain of an amplifier in the front end of the camera. Ifthe signal is too high, it is reduced by an amount to bring it down towithin the range, and if too low, the signal is increased by an amountthat brings it up to within the range. The amount of any reduction orincrease then becomes part of the data of the image that are recordedalong with the digital representation of the signal within the range.

The luminance values of the image signals over a wide dynamic range ofinterest are, in effect, shifted to levels within a smaller window ofluminance values that is defined by limitations of the image acquisitionand/or signal processing systems. The amount of the signal shift,preferably made in increments equal to the luminance range of thewindow, then also becomes part of the data representing the image. Thissignal adjustment may be made independently for each pixel of an image,or, alternatively, made the same for groups of adjacent pixels. Since,in a typical digital camera, an analog-to-digital converter translatesan analog signal from a photosensor into the digital domain, and inlarge measure sets the limited luminance value range window size, thesignal adjustment is preferably implemented, in such imaging systems, bycontrolling the gain of an amplifier at the input to theanalog-to-digital converter.

An image having the wide dynamic range of interest may then bereconstructed from the luminance value within the window and the amountof signal shift. In digital camera embodiments, portions of the imagehaving luminance levels outside of the defined window may bereconstructed from both the analog-to-digital converter output and thegain data rather than being saturated. Details in a portion of the imagethat would otherwise be saturated by the camera, and thus have a uniformwhite or black intensity, are now viewable. This is because the cameragain is adjusted to maintain the signal level within the range of thecamera digital processing paths. When forming an image from dataobtained from the camera digital processing paths, intensity values areadjusted by the amount of gain to which the amplifier in that path wasset when acquiring these data. Since a typical digital cameraautomatically sets the exposure based upon an average luminance of allor a portion of the image, there are typically only small amounts of theimage that need to be shifted and processed in this way.

As with standard cameras, it is desired to perform at least someprocessing of the image data before they are stored or otherwiseoutputted from the camera. But the representation of image pixelluminance levels by separate luminance and gain quantities is not whatcommonly used image processing algorithms have been designed to process.Examples are algorithms to de-mosaic the image, adjust image whitebalance and reduce image noise. If data of an image having the fulldesired dynamic range is first formed within the camera from theseparate quantities, these data may then be processed with suchalgorithms in the usual way. But the formation of images with such awide dynamic luminance range requires that the camera has the capabilityof handling them, which requires more processing, more memory andperhaps a wider system bus than included in the current typical digitalcamera. Therefore, in order to avoid this, the image processingalgorithms may be applied to just the data coming from theanalog-to-digital converter within the luminance range window. It hasbeen found to be unnecessary to employ the absolute luminance values ofthe pixels for processing the images with such algorithms. Luminancedata from contiguous pixels with the same associated gain values areprocessed together. After processing, the image pixel values arerepresented by the processed image data and their associated gainvalues.

The data from the analog-to-digital converter and the amplifier gain mayalso be used within the camera to generate data of a tone mapped image.Such an image has a dynamic range that is less than the wide dynamicrange that is described in the immediately preceding paragraph but doescontain details of the images in regions that would normally besaturated by the camera. Prior art tone mapped images have been formedfrom high dynamic range (HDR) images in order to display properly on amore limited dynamic range of a video monitor or other display device.Here, the tone mapped images are instead formed from high dynamic rangedata acquired within a low dynamic range camera. The tone mapped imagesmay then be processed and utilized by existing algorithms and devicesdesigned to work with the low dynamic range camera. But if it is desiredto obtain HDR images, the tone mapped image data may later be combinedwith the gain data. Images having either a high or a low dynamic rangemay thus be obtained from camera data, depending upon the devices usedto process the data or display the images, and the desire of the user.

In other embodiments described in the above-referenced application Ser.No. 11/467,993, the parameter is related to the duration of the exposurethat is controlled to maintain the signal within the range of thecamera. If the signal that would result from an exposure of a portion ofthe image for the full exposure time period is too high, a measurementis taken before the end of the exposure time period, while the signal iswithin the defined range. The time during the exposure at which theluminance value is measured, or the relationship of the exposure time tothe full exposure time period, then becomes part of the data of theimage that are recorded along with the digital representation of thesignal within the range. The luminance values within such an imageportion are then reconstructed by increasing the individual measuredluminance values by a factor proportional to the relationship of theactual exposure time to the full exposure time period. Typically, only asmall part of an image needs to be processed in this manner, since dataof most of an image are usually captured during the full exposure time.

The data acquired in these other embodiments, namely luminance valuesand corresponding time data, are similar in form to the data acquired inthe embodiments primarily described in the present application, whichinclude luminance values and corresponding gain data. The luminance andtime data form may therefore be processed and utilized in many of theways described herein for data in the form of luminance and associatedgain data.

It will also be noted that the data representation and processingtechniques described herein are not limited to use with luminance datafrom a photosensor but rather can also be applied generally to othercontinuums of magnitudes that are divided into discrete ranges, eitherbecause of device limitations, such as the limited bandwidth of thecameras described above, or for some other reason. The magnitudes may berepresented by values within the individual ranges plus anidentification of the ranges in which the values lie. The data withinthe individual ranges may be processed separately, and the processeddata of the individual ranges then combined with each other and with theidentity of the corresponding ranges in which the data lay beforeprocessing. These processes are described hereinafter, as examples, forimplementation with digital cameras.

Additional aspects, advantages and features of the present invention areincluded in the following description of exemplary examples thereof,which description should be taken in conjunction with the accompanyingdrawings.

All patents, patent applications, articles, books, specifications, otherpublications, documents and things referenced herein are herebyincorporated herein by this reference in their entirety for allpurposes. To the extent of any inconsistency or conflict in thedefinition or use of a term between any of the incorporatedpublications, documents or things and the text of the present document,the definition or use of the term in the present document shall prevail.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a camera or other video acquisition devicethat includes selective control of analog signal gain;

FIG. 2 is a flowchart that illustrates a first embodiment forcontrolling the saturation of acquired image data, which shows operationof the device of FIG. 1 to control its analog signal gain;

FIG. 3 illustrates types and grouping of image pixels as used by thedevice of FIG. 1;

FIGS. 4A and 4B are curves that illustrate examples of operation of thedevice of FIG. 1 when controlling the saturation of acquired image data;

FIG. 5 is a table of values generated by operation of the device of FIG.1 according to the flowchart of FIG. 2;

FIG. 6 is a circuit diagram that illustrates a second embodiment forcontrolling the saturation of acquired image data, an analog controlcircuit that may be included in the device of FIG. 1;

FIG. 7 is a circuit diagram that illustrates a third embodiment forcontrolling the saturation of acquired image data, a digital controlcircuit that may be included in the device of FIG. 1;

FIG. 8A illustrates a first embodiment of processing digital image datawithin the device of FIG. 1, and FIG. 8B shows a use of the dataprocessed according to FIG. 8A to form images;

FIG. 9A illustrates a second embodiment of processing digital image datawithin the device of FIG. 1, and FIG. 9B shows a use of the dataprocessed according to FIG. 9A to form images;

FIG. 10 shows an image having two saturated regions, such as representedby the curves of FIG. 4A;

FIG. 11A illustrates a third embodiment of processing digital image datawithin the device of FIG. 1, and FIG. 11B shows a use of the dataprocessed according to FIG. 11A to form images;

FIGS. 12A and 12B show two examples of the range of luminance of anobject scene that is reproduced as a tone mapped image; and

FIG. 13A illustrates a fourth embodiment of processing digital imagedata within the device of FIG. 1, and FIG. 13B shows a use of the dataprocessed according to FIG. 13A to form images.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Electronic Camera Example

In FIG. 1, an example of a camera in which the present invention may beimplemented is schematically shown, which may be a still camera or avideo camera. It includes a case 11, an imaging optical system 13, usercontrols and indicators 15 that generate and receive control signals 17,a video input-output receptacle 19 with internal electrical connections21, and a card slot 23, with internal electrical connections 25. Anon-volatile memory card 27 is removably inserted into the card slot 23.Data of images captured by the camera may be stored on the memory card27 or in an internal non-volatile memory (not shown). Image data mayalso be outputted to another video device through the receptacle 19. Thememory card 27 can be a commercially available semiconductor flashmemory, small removable rotating magnetic disk or other non-volatilememory to which image and/or video data can be written by the camera.

The optical system 13 can be a single lens, as shown, but will normallybe a set of lenses. An image 29 of a scene 31 is formed in visibleoptical radiation through an aperture 32 and a shutter 33 onto atwo-dimensional surface of an image sensor 35. A motive element 34 movesone or more elements of the optical system 13 to focus the image 29 onthe sensor 35. An electrical output 37 of the sensor carries an analogsignal resulting from scanning individual photo-detectors of the surfaceof the sensor 35 onto which the image 29 is projected. The sensor 35typically contains a large number of individual photo-detectors arrangedin a two-dimensional array of rows and columns to detect individualpixels of the image 29. Signals proportional to the intensity of lightstriking the individual photo-detectors are obtained in the output 37 intime sequence, typically by scanning them in a raster pattern, where therows of photo-detectors are scanned one at a time from left to right,beginning at the top row, to generate a frame of image data from whichthe image 29 may be reconstructed. The analog signal 37 is appliedthrough a variable gain amplifier 38 to an analog-to-digital convertercircuit chip 39 that generates digital data in circuits 41 of the image29. Typically, the signal in circuits 41 is a sequence of individualwords of digital data representing the intensity of light striking theindividual photo-detectors of the sensor 35.

The photo-detectors of the sensor 35 typically detect the intensity ofthe image pixel striking them in one of two or more individual colorcomponents. Early sensors detected only two separate colors of theimage. Detection of three primary colors, such as red, green and blue(RGB) components, is now common. Currently, image sensors that detectmore than three color components are becoming available.

Processing of the image data in circuits 41 and control of the cameraoperation are provided, in this embodiment, by a single integratedcircuit chip 43 (which may also include the analog-to-digital converterinstead of using the separate circuit chip 39). The circuit chip 43 mayinclude a general purpose processor that executes algorithms defined bystored firmware. These functions may be implemented by severalintegrated circuit chips connected together but a single chip ispreferred. In addition to being connected with the circuits 17, 21, 25and 41, the circuit chip 43 is connected to control and status lines 45.The lines 45 are, in turn, connected with the aperture 32, shutter 33,focus actuator 34, sensor 29, controllable gain analog amplifier 38,analog-to-digital converter 39 and other components of the camera toprovide synchronous operation of them. Signals in the lines 45 from theprocessor 43 drive the focus actuator 34 and set the size of the openingof the aperture 32, as well as operate the shutter 33. The gain of theanalog signal path may also set by the processor 43 through the lines45. Signal gain is typically controllable in the analog-to-digitalconverter which, in the case of a CCD sensor, is part of the sensor, orin the case of a CMOS sensor, is part of a separate analog-to-digitalconverter as shown in FIG. 1. Use of a separate controllable gain analogamplifier 38 increases the amount of control that is possible over thesignal gain in the analog front end (AFE) of the camera.

A separate volatile random-access memory circuit chip 47 is alsoconnected to the processor chip 43 through lines 48 for temporary datastorage. Also, a separate non-volatile memory chip 49, connected to theprocessor chip 43 through lines 50, may be included for storage of aprocessor program or firmware, calibration data and the like. The memory49 may be flash memory, which is re-programmable, or a memory that isprogrammable only once, such as a masked programmable read-only-memory(PROM) or an electrically programmable read-only-memory (EPROM). A usualclock circuit 51 is provided within the camera for providing clocksignals to the circuit chips and other components. Rather than aseparate component, the clock circuit for the system may alternativelybe included on the processor chip 43.

A source 53 of artificial illumination, such as a flash Lamp or othersource of light pulses, is preferably built into the camera case 11. Thesource 53 operates in response to control signals from the processor 43through control lines 55. The light source 53 may be a xenon flash lampor a white light-emitting-diode (LED). The processor 43 preferablycontrols the timing and other aspects of the light source 53.

The controllable gain amplifier 38 may be included as part of the sensor35, part of the analog-to-digital converter 39, or as a circuit that isseparate from these, which is what is shown. The gain of the amplifier38 is typically controlled by the processor 43 as one of severalparameters that are automatically set prior to taking a picture of anobject scene illuminated with a given light level. Other such parametersinclude the duration of the exposure (shutter speed) and the size of theaperture opening. Additionally, the gain of the amplifier is adjustedherein to effectively expand the dynamic range of the system in order tominimize or avoid saturation of the image. Positive or negativesaturation, or both, are controlled in this way, as explained below withrespect to specific embodiments.

Image data acquired by a digital camera such as illustrated in FIG. 1are typically processed to compensate for imperfections of the cameraand to generally improve the quality of the image obtainable from thedata. The correction for any defective pixel photodetector elements ofthe sensor is one processing function that may be performed. Another iswhite balance correction wherein the relative magnitudes of differentpixels of the primary colors are set to represent white. Compensation ofthe image data for noise and variations of the camera optical systemacross the image and for variations among the sensor photodetectors mayalso be performed. Other processing typically includes one or more ofgamma correction, contrast stretching, chrominance filtering and thelike.

This processing may also include de-mosaicing the individual pixel datato superimpose data from spatially separate monochromatic pixeldetectors of the sensor 35 to render superimposed multi-colored pixelsin the image data. Such a sensor is in common use, is known as Bayersensor. De-mosaicing combines separate red, green and blue outputs ofadjacent detectors to form signals of individual pixels of the image.De-mosaicing then makes it desirable to further process the data toenhance and smooth edges of the image. Another type of available sensorhas individual sensors for the primary colors superimposed, known as theFoveon sensor, so de-mosaicing processing of its output data isunnecessary. Another sensing technique is to use a sensor with a singleset of broad light spectrum photodetectors, one for each pixel, and thenposition a rotating color wheel in front of the sensor. The individualphotodetectors then receive the multiple primary colors in sequence.De-mosaicing processing is also unnecessary with this approach.

The processed data are then usually compressed within the camera by useof a commercially available algorithm before storage in a non-volatilememory.

First Saturation Control Embodiment Digital Processing

In one embodiment of a technique that controls the amount of saturationwithin an image, data of two images of a scene are acquired in rapidsuccession. Both images are preferably taken with the same exposureparameters, such as exposure duration, the size of the aperture and thegain of the signal path. These are typically set automatically withinthe camera or other video data acquisition device in response tomeasurement of at least the luminescence of the scene that is made bythe camera. The first image is used to determine whether any areas ofthe image are saturated, and, if so, this information is utilized to setthe amount of signal gain within the camera that is used to acquire dataof a second image in these areas. The gain is set to a level thatreduces, or even eliminates, the effects of the saturation encounteredin the first image. It is the second image that is recorded. Data of thefirst image may be discarded after they have been used to set the gain.The first image may be a reduced resolution preview image and the secondimage a full resolution image whose data are retained for later viewingof the image of the scene. More than one preview image may be acquired,although use of the preview image taken immediately prior to the finalimage is preferred for saturation correction. Two or more preview imagesmay be taken in order to perform other functions, such as setting theexposure parameters for taking the final image, correcting for motion ofthe camera within the scene being photographed or to control the flashlamp.

Referring to FIG. 2, a process that may be implemented by the signalprocessor 43 (FIG. 1) to correct for image saturation is described. Whenpower is applied to the camera, it acquires data of a succession ofpreview images until the shutter button is pressed by the user, at whichtime information from the most recently taken preview image is used toset the camera for recording the final image with saturation correction.The act of acquiring data of a preview image is indicated at 61 in FIG.2.

FIG. 3 shows some of the pixels of an image frame, representedindividually by square dots. Data of the pixels are typically obtainedwithin the sensor 35 (FIG. 1) for a full resolution image by scanningacross a row, then proceeding to an adjacent row, and so on until dataof all the pixels of the frame have been acquired. But rather than usingthe data of all the pixels from the preview image to identify itssaturated regions, a lesser amount of data is preferably utilized torepresent a preview image of reduced resolution. This reduces the amountof processing that is necessary to identify saturated regions andtherefore takes less time to perform.

There are two ways illustrated in FIG. 3 to reduce the amount of imagedata. One is to acquire data of only a few of the pixels that are evenlydistributed over the image. Pixels 63 and 65 are examples. Data areacquired for every eighth pixel along a row, and this is done for everyeighth row across the image frame, in this example. The data of onepixel are taken to represent the value of each pixel within a block thatincludes the one pixel. Alternatively, data of all the pixels within adefined area, such as those within each of adjacent blocks 67 and 69,may be averaged to come up with a single set of values that are used torepresent all the pixels within a block. Each block contains 64 pixelsin this specific example. This is done for each block across the image,and the average values are then used for processing to determine whethersaturation exists.

Returning to FIG. 2, it is determined at 71, for a first block ofpixels, whether its magnitude (either for one central pixel or anaverage of all pixels in the block) is at a maximum or minimum of arange of signal levels that can be handled by the analog-to-digitalconverter 39, the bus 41 and/or the signal processor 43. If so, it ishighly likely that the value for that block is outside the range ofvalues that can be processed, thereby in saturation, and therefore doesnot represent the true magnitude of the image in that block. If thesignal value for the block is at a minimum, a +1 gain value istemporarily stored in a camera memory for that block, as indicated at73. This indicates that the gain of the amplifier 38 (FIG. 1) is to beincreased by a +1 increment when data of the pixels in that particularblock are subsequently acquired as part of the final image. If thesignal value for the block is at a maximum, on the other hand, a −1 gainvalue is stored, at 75. If the block signal value is within the range,either nothing need be done or a 0 indication may be stored.

This process is performed for each block of pixels across the previewimage frame. After the functions 71, 73 and 75 have been completed forthe first block, it is determined at 77 that they need to be performedfor the next block in order, so process flow returns to 71. This cycleis performed until data of any saturation are acquired for all theblocks across the image frame. If the shutter is not depressed, asindicated at 79, the process returns to 61 to acquire data of anotherpreview image. The same processing of the next preview image isperformed as described above. Saturation data will typically be retainedonly for the most recent preview image, for use after the user hasdepressed the shutter to capture data of a full resolution image that isto be corrected.

Reference is made to FIG. 4A to illustrate image saturation and itscorrection. Two segments 81 and 83 of a signal from the sensor 35(FIG. 1) of a single image frame are show. The segment 81 goes intonegative saturation when it falls below a bottom threshold 85 of anormal range of signal magnitudes to which the processing path islimited by the analog-to-digital converter 39, the bus 41, the signalprocessor 43, all of them, or perhaps some other bit width limitedcomponent of the system. Similarly, the segment 83 becomes positivelysaturated when going above a maximum level 87 of the normal range. Afterthe analog-to-digital converter 39, a binary signal carried by the bus41 and within the signal processor 43 has a number of bits representingthe normal range of the image signal within a selected resolution. Sincethe data paths and processing can typically handle only the normalrange, any values below or above that range will be represented by arespective one of the lower limit 85 or higher limit 87. Any detail inthese portions of the image is thus lost.

But as a result of the image saturation processing being describedherein, a saturated portion of the image signal is shifted in magnitude,by appropriately adjusting the gain of the amplifier 38, back into thenormal range. The value of this shift is indicated in FIG. 4B, wherein acurve 91 takes on a +1 value when the signal is negatively saturated,and a curve 93 a −1 value when the signal is positively saturated. Theseare the gain adjustment values determined, on a pixel-block bypixel-block basis, at 73 and 75 of the process of FIG. 2. They may betemporarily stored in any convenient manner, a table of such valuesbeing shown in FIG. 5, where an entry exists at least for every pixelblock across the preview image that is positively or negativelysaturated.

These stored values are then accessed as a result of the user pressingthe shutter button of the camera, at 79 of FIG. 2. Since a number ofsuccessive pixels in a row lie in the same block of the preview image,eight in the example of FIG. 3, it is preferable to acquire and processthe data in groups of that number of pixels since any necessary gainadjustment will be the same for each pixel of this group. Therefore, at97 of FIG. 2, a first series of pixels that lie in a row across one ofthe blocks of FIG. 3 are accessed. At 99 of the process illustrated inFIG. 2, it is determined, by access to the table of FIG. 5 or otherwise,whether there is gain adjustment value stored for those pixels. If thereis a +1 gain adjustment, the gain of the amplifier 38 (FIG. 1) isadjusted upwards, at 101. If there is a −1 gain adjustment recorded, thegain is adjusted downwards, at 103. If no gain adjustment value existsfor the series of pixels, then no gain adjustment is made.

The magnitudes of the pixels in the identified series are then acquiredat 105 after any gain adjustment. This shifts the magnitudes ofsaturated pixels back into the normal range of the camera signalprocessing. They are therefore no longer saturated. Detail is restoredin the portions of the image that were saturated. The processor 43 keepstrack of the gain adjustment values along with the data of the magnitudeof each pixel, as indicated at 107, so that the gain adjusted magnitudesmay be restored back to their actual values when an image isreconstructed from the acquired data.

This process is performed for the pixels of each series within thepreview image blocks. At 109, it is determined whether there are anypixels of the image frame remaining for which data have not yet beenacquired. If so, the next series of pixels are selected at 111 and theprocess beginning with 97 is repeated for them. When data have beenacquired for all the pixels of the image frame, the process returns to61 to acquire and process another preview image in the manner describedabove.

The process of FIG. 2 has been described to correct for both positiveand negative saturation. However, correction for only positivesaturation may be preferred for many imaging applications. This has theadvantage that the gain adjustment may be represented by a single databit. Also, negative saturation correction will typically result in theamount of noise in an image to increase in the saturated portion as aresult of increasing the signal amplification.

Note that the process implemented by signal processor 43 of FIG. 1 mayemploy preview images, reduced resolution preview images, or fullresolution temporarily stored or pre-captured images. The latter,although memory and processing intensive, may be desirable for imageswith large amounts of detail, where a wide image data dynamic rangeneeds to be preserved for closely spaced pixels that are alternatelynear negative saturation, below the range of the digital camerasanalog-to-digital converter, and near positive saturation, above therange of the digital camera's analog-to-digital converter. Additionally,it may use image data from a single image, the image to be captured, byanalyzing 1 or more pixels preceding a pixel to be captured, andadjusting the gain for the pixel to be captured accordingly.

Second Saturation Control Embodiment Analog Control

As an alternative to the saturation correction process described withrespect to FIG. 2, hardware elements may be added to the device systemof FIG. 1. An analog example is shown in FIG. 6, where the elements thatare the same as those of FIG. 1 are identified by the same referencenumbers. Rather than the processor 43 determining and setting the gainof the amplifier 38 for saturation correction, a comparator 121 is addedto the system. One input of the comparator 121 receives the sensoroutput analog signal in line 37. Another input 123 is connected with areference voltage that defines a positive saturation threshold, such asthe threshold 87 of the example of FIG. 4A.

When the signal in the line 37 exceeds the positive saturation thresholdin the line 123, an output 125 of the comparator changes state andcauses the gain of the amplifier 38 to decrease by the unit of −1. Thecomparator 121 is preferably provided with hysterisis so that its outputswitches back to 0, thereby restoring the gain of the amplifier 38 toits original value, at a lower threshold than that which caused it'soutput 125 to switch from 0 to −1. The 0 or −1 output of the comparator121 provides the gain adjustment value that is sent to the processor tobe associated with the values of the signal in the bus 41 acquired withthose gain adjustments. The combination of these two provides theabsolute value of the image signals.

If it is desired to also correct for negative saturation, a secondcomparator may be added with an output connected to the amplifier 38 toincrease its gain when the signal 37 falls below a second thresholdvoltage applied to this second comparator. The output of this secondcomparator is then also connected with the signal processor 43.

The gain of the amplifier 38 is shown in FIG. 6 to be adjusted tocompensate for saturation. This amplifier gain will commonly alsocontinue to be adjusted by the signal processor 43 through lines 45(FIG. 1) to set particular image exposure parameters.

It will be noted that the circuit of FIG. 6 adjusts for saturation inreal time. There is no need to previously acquire and process previewimages for this purpose, although preview images may be used todetermine exposure parameters or for other purposes.

Third Saturation Control Embodiment Digital Control

A real time system may alternatively be implemented digitally, anexample of which is shown in FIG. 7. This digital circuit is preferablyincluded as part of the signal processor 43 but can alternatively beimplemented separately. A digital comparator 129 receives the output 41of the analog-to-digital converter 39, and compares it with a digitalvalue 131 that defines the positive saturation threshold and,optionally, a digital value 133 that defines the negative saturationthreshold. An output 135 of the comparator 129 carries the 0, +1 and −1gain adjustment values that are applied to the amplifier 38 to controlits gain. There will typically be a one pixel delay between thecomparator 129 detecting that the signal in bus 41 has crossed over oneof the thresholds and an output 135 that adjusts the gain of theamplifier 38. For example, when a value on the bus 41 of a pixel crossesover the positive saturation threshold 131, the gain of the amplifier 38is not adjusted to affect the value of that pixel but is adjusted intime to adjust for acquisition of data of a later acquired pixel. Thedelay may be the time to acquire data of one or more pixels.

The output 135 is also sent to a core processing unit 137 of theprocessor 43 for association with the signal value output 41 of theanalog-to-digital converter. The gain adjustment values used to controlthe gain during acquisition of the magnitude of individual pixels areassociated with those pixels. The absolute values of the pixels areprovided by a combination of the two.

The gain of the amplifier 38 is shown in FIG. 7 to be adjusted tocompensate for saturation. This amplifier gain will commonly alsocontinue to be adjusted by the signal processor 43 through lines 45(FIG. 1) to set particular image exposure parameters.

It will be noted that the circuit of FIG. 7 adjusts for saturation inreal time. There is no need to previously acquire and process previewimages for this purpose, although preview images may be used todetermine exposure parameters or for other purposes.

Although the above embodiments have been described with just twothresholds, one for negative saturation and one for positive saturation,the invention is not so restricted. Multiple thresholds, both in thenegative and positive directions from a nominal level, may be employedto provide finer control of gray scale renditioning, and the potentialof reconstructing a final image with a wider and smoother dynamic range,free of gray and/or color level quantization step aberrations, oftencalled contouring. The use of multiple thresholds may also cause thenumber of bits representing the gain data to increase.

Further Processing and Use of the Image Signal Data

In each of the three alternative embodiments described above withrespect to FIGS. 2, 6 and 7, the character of the image signals receivedby the camera processor is the same. In each, the value of a particularpixel is represented by the data output of the analog-to-digitalconverter 39 plus data representing the gain setting of the analogamplifier 38 when data of that pixel were captured. Typically, if theamplifier gain remains at its default level, no gain data aremaintained, or only a small amount of data are maintained, thus keepingthe amount of gain data low.

With reference to FIG. 8A, one embodiment is described for storing thesedata in a raw form in the memory card 27 without image processing withinthe camera, or output there raw data for some other use. The lines 41carry the digital output of the analog-to-digital converter 39, and thelines 143 the bit(s) that represent the setting of the gain of theamplifier 38. The image data in lines 41 may optionally be processedwith one or more of the conventional image processing algorithmsmentioned above, as may be appropriate without use of the associatedgain data in lines 143 but the storage of the raw acquired data withoutimage processing is shown. The image luminance data are maintainedseparate from the amplifier gain data, as indicated at 147.

The data at 147 may be output from the camera in raw form but moretypically are compressed within the camera, as indicated at 149, beforebeing stored in the memory card 27 or output from the camera in someother way. The associated gain data may be separately stored on thememory card 27 without compression since the amount of gain data willtypically be quite small, while the much larger amount of imageluminance data are compressed. But if both are compressed, the popularJPEG algorithm may be used, for example. If so, the gain data may becarried in a “sub-band” where the image data are in a standard JPEGformat of three primary colors such as red, green and blue. The imageand gain data are then compressed and stored in a manner that allowsde-compression with the corresponding standard JPEG algorithm. In thiscontext “sub-band” refers to an ancillary storage location within animage file format structure. This storage location could be physicaldigital data space that is allocated for extra data that is related tothe main imaging data, or could be incorporated into the main imagingdigital data itself by the use of, for example, stegographic techniques.For the present invention, this “sub-band” carries gain data, that canbe interchangeable referred to as “sub-band data, “ancillary data”,“side chain data” or “support data”.

Also note that although the above description calls for both the imageand gain data to be compressed, as does the accompanying FIGS. 8A and8B, compressing of one or both of these data objects is not required tocarry out the techniques being described. Traditionally, digital cameraimagery is compressed using the JPEG compression and file formats.However, since the addition of gain data for the purpose of high dynamicrange image reconstruction is unique to the present invention, no suchcompression need be considered for gain data “compatibility”. Such isnot the case for digital image data compatibility, for virtually allcommercial digital camera and image processing software implementationsare either based on the JPEG image format, or are capable of reading,and/or operating on, an image file adhering to the JPEG standard.

Referring to FIG. 8B, use of this stored data to reconstruct orotherwise utilize the images is shown. The data read from the memorycard 27 are decompressed, as indicated at 151. If a JPEG algorithm isused, it is fully decompressed to include the gain data. The image andgain data are then separately available, as indicated at 153. These dataare then combined, at 155, to provide image data with a wide dynamicrange. The combination of data indicated at 155 involves shifting thevalue of image data of an individual pixel or group of pixels by thevalue of any gain data for that pixel or group. For most pixels of animage, there is no such shift since most pixels of a typical picture arenot saturated when the camera amplifier 38 is at its normal defaultgain. The amount of gain data is therefore typically quite small. Butfor those pixels where the amplifier gain was changed to avoid theeffects of saturation, gain data for those pixels are maintained andused at 155 to shift values of their image data. The dynamic range ofthe image data is therefore selectively expanded from what the cameracan normally provide, but only in portions of the image where necessary.The amount of extra data that are stored and processed is therefore keptlow.

Prior to displaying, printing, storing or otherwise using the fulldynamic range data at 157 of FIG. 8B, the image data may be processed at156. Since the luminance image data and associated gain data arecombined at 155, the high dynamic range data may be processed withstandard image processing algorithms. These algorithms are moretypically executed within the camera but, in the example of FIG. 8A, theimage and gain data are not combined in a manner that allows executionof these standard image processing algorithms so are, in this example,postponed until post-processing in a computer or the like, where thehigh dynamic range data have been reconstructed from the image and gaindata.

FIGS. 9A and 9B illustrate a different way to process and utilize HDRimage data than shown in FIGS. 8A and 8B. Image and gain data 159 arecombined at 161 to generate data of a HDR image. These full dynamicrange image data include data of image locations that have not beenshifted in gain, plus image data acquired from regions of the imagewhere the gain was changed in order to avoid saturation after theirluminance range is shifted according to the gain values associated thoseimage data. These image data may then be processed at 163 with thestandard image processing algorithms described above. Prior to storagein the memory card 27, or alternatively to some other output of thecamera, the processed HDR image data may be compressed, at 165.Alternatively, the uncompressed processed HDR image data may be storedor output.

The use of the camera output data from FIG. 9A is straight forward, asshown in FIG. 9B. A processor in a personal computer or some other hostutilization device first decompresses the stored data, as indicated at167, if the data have been stored in a compressed form. The HDR imagedata at 169 result, and thus may be viewed, printed or utilized in someother manner, as indicated at 171.

A primary advantage of the technique illustrated in FIG. 9A is that thedata after 161 represent an image frame within the camera in a fulldynamic range with a single set of data. This is therefore similar tohow existing cameras represent image data, except that the dynamic rangeis considerably expanded. But the cost of providing this ability is thateach image is represented by a large amount of data, which increases theinternal memory requirements and amount of processing that needs to beperformed. A camera utilizing the technique of FIG. 9A still has theadvantage, as does the technique of FIG. 8A, of using a standard lowcost analog-to-digital converter 39 (FIG. 1), such as one outputting8-bits, but with the ability to generate the same HDR image data that a12-bit, 16-bit, 24-bit or larger A/D converter 39 could provide, if thegain factor of the amplifier 38 is used in the manner described above.

De-mosaicing, one of the standard image processing techniques used incameras when the photosensor 35 (FIG. 1) is a type that uses the Bayergrid, may be performed directly at 163 of FIG. 9A. Of course, if thephotosensor 35 is the Fovion type, the image data does not then need tobe de-mosaiced. In addition to the examples of de-mosaicing, whitebalance and noise reduction mentioned above, other image processing andenhancement may be performed at 163 as is typically carried out incameras or other image acquisition devices. Other processing algorithmsthat may be implemented at this stage include those for digitallighting, color mapping and highlight recovery.

FIG. 10 shows an example image frame 173 that has a region 175 thatwould otherwise be positively saturated, so thus has a negative gainadjustment, and region 177 that would otherwise by negatively saturated,so thus has a positive gain adjustment. In this case, it is only the rawimage data of the regions 175 and 177 that are acquired after making anadjustment in the camera gain. Although only two saturated portions ofthe image are illustrated in FIG. 10, images may have more such distinctareas, or only one. But the total amount of area of a typical image thatis saturated will usually be quite low since the camera automaticallysets the exposure level to an average luminance of the image beingrecorded.

Data of the gain adjusted regions of the image, such as regions 175 and177 of FIG. 10, can be processed separately from the data of the rest ofthe image that do not have associated gain adjustment data. Theprocessed data of all the different image regions are then combined toproduce an image data set containing dynamic range information that isgreater than that of the system's intrinsic dynamic range. This separateregional processing approach may be applied to processing actions suchas de-mosaicing, white balance compensation, digital lightingcompensation, noise reduction and highlight recovery. This approach isdifferent than that those used in the example of FIGS. 8A and 8B, wherethe image processing 156 is performed in post processing, and in theexample FIGS. 9A and 9B, where the image processing 163 is performed inthe camera, both after the image and gain data have been combined. Inthe technique being described with respect to FIG. 10, on the otherhand, data of the separate regions are processed without first having toreconstruct data of a full dynamic range image. The gain adjustment dataassociated with the image data of these distinct regions are generallynot necessary to carry out such image processing. It is only necessary,when using this image processing technique, that the data of eachdifferent gain adjusted region of the image be independently processed.

If an image processing algorithm applied on a regional basis, asdescribed with respect to FIG. 10, operates on blocks of pixel data,some special processing may be necessary for data of the edges of thedistinct regions in order to knit the image portions together withoutthe boundary appearing in the image. An appropriate interpolationbetween data on opposite sides of the boundary between adjacent regionsmay be performed as part of executing an algorithm that utilizes suchdata. But it is preferred that the image data be combined with theirassociated gain data for a few pixels on either side of the boundary, asnecessary for execution of the image processing algorithm, which thenuses this full dynamic range data in the limited boundary region. Thegain data are used by the image processing for only this limitedpurpose. This does therefore increase the amount of data that the cameramust process but not significantly. The internal bus width, theprocessing capability or internal memory need not be expanded. A doubleprecision memory may be used, and the high dynamic range data ofindividual pixels represented, for example, by two 8-bit words in an8-bit wide processing system of the camera.

The image data processed for each region are then combined with theirassociated regional gain data for additional processing steps. Suchadditional processing steps include, but are not limited to, image tonemapping, conversion to a standard JPEG format image, and the creation ofa full dynamic range HDR image. The use of image data processed in thisway with image tone mapping are described below with respect to furtherexamples of FIGS. 11A and 13A.

FIG. 11A illustrates a different technique for the further processing ofdata in the camera than do FIGS. 8A and 9A. Rather than storing theimage data and gain data separately, data of tone mapped images arecalculated at 183 from both the image data and the gain data. In thisexample, image processing as described above with respect to FIG. 10 isfirst performed at 181 before data of tone mapped images are performed.Images formed from the data of the toned mapped version will have lessdynamic range than those from the HDR image data of FIGS. 8B and 9B butthey are viewable with details of any regions that would otherwise besaturated being visible. Tone mapping is widely used in computergraphics and other imaging applications. Tone mapped image data aregenerated at 183 by adjusting the values of individual pixels in theincoming image data 41 proportionately across a defined maximum dynamicrange. The determination of the individual pixel values includes use ofany gain data in the lines 143 that are associated with those pixels.The gain value for a pixel affects where its value lies in the defineddynamic range of the tone mapped image.

Although displaying more image details in the highlight and shadowregions of the image, the defined dynamic range of the tone mapped imageneed be no greater than the dynamic range of the acquisition systemwithout the gain adjustment being made, and will be less than theresulting dynamic range of the HDR data of FIGS. 8B and 9B. The pixelsof the tone mapped images may then be represented by the same number ofbits as the output of the A/D converter 39 (FIG. 1) and thereby carriedby the same internal bus and processed in the same manner within theprocessor 43. This is illustrated in FIG. 12A, wherein a luminance range(vertically) 197 of a tone mapped image is the same as a luminance range199 that the number of bits of the output of the A/D converter 39 canexpress. That A/D range is effectively tripled in the illustration ofFIG. 12A by a −1 gain adjustment value shifting this window upward to201 to capture data of any otherwise positively saturated image regions,and downward to 203 by a +1 gain adjustment to capture data of anyotherwise negatively saturated image regions.

The dynamic range portion 199 is reduced somewhat to a portion 205 ofthe dynamic range 197 of the tone mapped image by proportionatelyadjusting the values of luminance of the pixels of the image within theluminance range 199 to fit within the range 205. The range 205 is chosento be greater than fifty percent of the total image dynamic range 197,since the vast majority of the image area will typically be within thisrange, and less than ninety percent, in order to leave some range fordetails of any image region(s) that would otherwise be saturated. Aportion 207 of the tone mapped image carries the luminance variations ofthe range 201 of the luminance from the object scene, and a portion 209carries the luminance variations of the range 203.

Although the examples given so far adjust the camera system gain by onlyone or two increments, this has been to make the operational principleseasier to explain and understand. A typical system will have more thantwo adjustable gain increments. FIG. 12B shows four such increments, twopositive and two negative. At the high end, the gain adjustment can be,for example, represented by 8-bits which provides up to 256 differentgain adjustment values. FIGS. 12A and 12B illustrate the characteristicsof tone mapped images that make their generation and use particularlyadvantageous for digital cameras.

The form of the data obtained according to FIG. 11A, as indicated at185, is the tone mapped image data alone. The gain adjustment data neednot be stored in the memory card 27 or otherwise output from the memory,in this example. The tone mapped data 185 are then typically compressedat 187 before storage on the memory card 27. A standard JPEG compressionalgorithm may be used on the tone mapped image.

There are several advantages of processing and storing data in themanner described with respect to FIG. 11A. One is that the tone mappedimage may be utilized and displayed as the final image, particularly ondevices having limited dynamic range. No special processing is necessaryto view such images. Such an image is compatible with current cameras,displays, printers and other image data processing equipment. A displayon the camera may use the tone mapped image data to display imageswithout any significant amount of additional processing.

FIG. 11B illustrates use of the stored tone mapped image data directly.The tone mapped image data are read from the memory card 27 anddecompressed at 191. The tone mapped image data are then obtained bythemselves, as indicated at 193, and are utilized by a display, printeror some other device 195. Good quality images within a defined dynamicrange are provided. Details are then visible in any areas of the imagethat would otherwise have been a uniform white or black because ofsaturation.

Another advantage of the tone mapped image is that an image with fulldynamic range may be reconstructed from the tone mapped image data whenthe gain data are also used, if this is desired. HDR images may then bedisplayed or otherwise utilized with equipment that itself has a widedynamic range. FIG. 13A shows the processing and storage of both thetone mapped data and the gain data, as a variation of the processingshown in FIG. 11A. The difference is that the data 185′ includes thegain data, and the compression 187′, if used, will take that intoaccount. Whether or not the separate gain data are compressed, they may,in most cases, be carried within a sub-band of a JPEG compressionalgorithm. This is because, as previously stated, most image pixels havebeen adjusted to fall within the digital camera's window of itsanalog-to-digital converter, bus width and processor. Therefore, mostimage pixels will typically not be associated with separate gain data,and the amount of gain data generated is therefore relatively small. Thecompressed data are then stored on a memory card 27 or output from thecamera in some other manner.

But if the gain data are read from the memory card 27 and used, widedynamic range image data may be obtained, as illustrated in FIG. 13B.The stored data are fully decompressed, at 191, resulting in the dataindicated at 193′. At 211, the tone mapped image data and gain data areprocessed together to provide data of the image over a wide dynamicrange, which are then displayed or otherwise utilized at 213. Theprocessing 211 may first reverse the processing 181 of FIG. 13A thatgenerated the tone mapped image data. The original image data and gaindata are then obtained. These data may then be combined in the samemanner as in 155 of FIG. 8B.

Processing and Use of a Similar Form of Image Data

In the embodiments described in the above-referenced application Ser.No. 11/467,993, the dynamic range of an image capturing system isexpanded by representing the luminance of individual pixels with a valueof the luminance outputted by individual photosensor elements plus theduration to which the photosensor element was exposed that resulted inthat luminance value. Since the average luminance of an image is set indigital cameras and other image capturing devices to be within thenon-saturated response region of its photosensor, most elements of thephotosensor are exposed for the same maximum time without becomingsaturated. The luminance values of most image pixels are simply theoutput at the end of the maximum exposure time. But when the photosensorelement becomes saturated before the end of the maximum exposure time,the output measured at an earlier time before the element becamesaturated is used to extrapolate to final value that is beyond thesaturated value. The actual luminance striking the photosensor elementis calculated in this manner.

This technique allows detail to be visible in positively saturatedportions of images that would otherwise be reconstructed as all white.An image is represented by the outputs of all the photosensor elementsplus, in the case of those that become saturated, a representation ofthe time less than the maximum exposure time at which the output wascaptured. In a specific example, the measured luminance value isincreased beyond the saturated level by being multiplied by a ratio ofthe maximum exposure time divided by the actual exposure time. Thisratio, or a corresponding quantity, is a parameter that is recordedalong with the measured luminance values to represent the image.

Image data in this form may be processed and utilized in ways similar tothe examples described above with respect to FIGS. 8A-13B. In thoseexamples, images are represented by luminance data plus a gainparameter. The luminance data are often initially processed bythemselves and then later combined with the associated gain data.Similarly, those same techniques may be used with image data in theformat described in aforementioned application Ser. No. 11/467,993,namely luminance data plus an exposure time parameter. Indeed, theexposure time ratio defined immediately above has many of the sameattributes as the gain parameter utilized in the embodiments describedearlier. In one case, the actual image is reconstructed in its portionsthat would otherwise be saturated by multiplying the measured luminancevalue by the gain, and in the other by multiplying the measuredluminance value by the exposure time ratio.

Conclusion

Although the various aspects of the present invention have beendescribed with respect to exemplary embodiments thereof, it will beunderstood that the present invention is entitled to protection withinthe full scope of the appended claims.

It is claimed:
 1. A method of processing electronic signals of at leastone optical image that has a dynamic range of magnitudes outside of apredefined range of magnitudes, comprising: acquiring preview image datahaving a reduced resolution that is less than a resolution of the atleast one optical image, wherein the preview image data is acquired toset gain for acquiring image signals of the at least one optical image;adjusting magnitudes of the image signals outside of the predefinedrange by quantities selected to allow measurement of the adjustedsignals within the predefined range, the quantities selected based onpreview image data acquired prior to the image signals, representing theimage signals by quantities representative of magnitudes of the imagesignals within the predefined range, and magnitudes of the adjustedsignals within the predefined range along with their associatedadjustment quantities, calculating data for a tone mapped image of theat least one optical image, by a processor, from the quantitiesrepresentative of the magnitudes of the image signals within thepredefined range and the magnitudes of the adjusted signals within thepredefined range and their associated adjustment quantities, andoutputting, by the processor, data of the tone mapped image via anoutput, wherein the data of the tone mapped image is further generatedbased on data generated by the analog-to-digital converter and dataassociated with photo exposure time of the image signals.
 2. The methodof claim 1, additionally comprising reconstructing at least one imagefrom the quantities representative of the magnitudes of the imagesignals within the predefined range, and the magnitudes of the adjustedsignals within the predefined range along with their associatedadjustment quantities.
 3. The method of claim 1, additionally comprisingstoring in a non-volatile memory the quantities representative of themagnitudes of the image signals within the predefined range, and themagnitudes of the adjusted signals within the predefined range and theirassociated adjustment quantities.
 4. The method of any one of claims1-3, wherein the associated adjustment quantities include a gain of theelectronic signals.
 5. The method of any one of claims 1-3, wherein theassociated adjustment quantities include quantities related to a time ofexposure.
 6. The method of claim 1, additionally comprising displayingan image from the tone mapped image data.
 7. The method of claim 1,additionally comprising reconstructing data of an image having a widedynamic range by combining the tone mapped image data with theadjustment quantities associated with the adjusted signals.
 8. Themethod of claim 7, additionally comprising displaying an image from thereconstructed data.
 9. The method of claim 1, wherein the predefinedrange of magnitudes is defined by a limitation of a system on which thesignal processing takes place.
 10. The method of claim 1, additionallycomprising, prior to adjusting magnitudes of the signals, determiningwhether the magnitudes of the signals are outside the predefined range,and, if so, thereafter determining their associated adjustmentquantities.
 11. The method of claim 10, wherein determining whether themagnitudes of the signals are outside the predefined range of magnitudesincludes referencing stored signals of a prior image.
 12. The method ofclaim 11, wherein the stored signals of a prior image include storedsignals of preview images having a resolution that is less than aresolution of the magnitudes of the adjusted signals within thepredefined range.
 13. The method of claim 10, wherein determiningwhether the magnitudes of the signals are outside the predefined rangeof magnitudes includes referencing signals of one or more pixels withinthe same image.
 14. The method of claim 10, wherein determining whetherthe magnitudes of the signals are outside the predefined range ofmagnitudes includes comparing the magnitudes of the signals with atleast one reference magnitude.
 15. The method of claim 1, wherein thesignals that represent the magnitudes of individual pixels of theoptical image and the adjustment quantities are associated with theindividual pixels.
 16. The method of claim 1, wherein the signals thatrepresent the magnitudes of individual pixels of the optical image andthe adjustment quantities are associated with blocks that individuallyinclude a plurality of individual pixels.
 17. The method of claim 1,additionally comprising processing the images by utilizing at least oneimage processing algorithm with magnitudes of the image signals withinthe predefined range, and magnitudes of the adjusted signals within thepredefined range, and thereafter combining the results.
 18. The methodof claim 17, additionally comprising thereafter employing the associatedpredefined range adjustment quantities with the combined results. 19.The method of claim 17, wherein combining the results of the processesincludes employing the associated predefined range adjustmentquantities.
 20. The method of claim 17, wherein the at least one imageprocessing algorithm includes a de-mosaicing algorithm.
 21. The methodof claim 17, wherein the at least one image processing algorithmincludes a white balance algorithm.
 22. The method of claim 17, whereinthe at least one image processing algorithm includes a digital lightingalgorithm.
 23. The method of claim 17, wherein the at least one imageprocessing algorithm includes a color mapping algorithm.
 24. The methodof claim 17, wherein the at least one image processing algorithmincludes a noise reduction algorithm.
 25. The method of claim 17,wherein the at least one image processing algorithm includes a highlightrecovery algorithm.
 26. The method of claim 17, wherein the at least oneimage processing algorithm includes a tone mapping algorithm.
 27. Themethod of claim 17, wherein the at least one image processing algorithmincludes a JPEG compression algorithm.
 28. The method of claim 17,wherein the at least one image processing algorithm includes a HDR imagereconstruction algorithm.
 29. A method of acquiring data of at least oneimage, comprising: acquiring preview image data having a reducedresolution that is less than a resolution of data for the at least oneimage, wherein the preview image data is acquired to set gain foracquiring image signals of the image; directing the image onto aphotosensor that generates an analog signal representing magnitudes ofpixels within the image; passing the photosensor analog signal through acontrollable gain amplifier that provides an output of a levelcontrolled analog signal; applying the level controlled analog signal toan analog-to-digital converter having a defined input range window andwhich outputs digital signals representing magnitudes of the levelcontrolled analog signal within the window; determining whether thelevel controlled analog signal has a magnitude outside the window, and,if so, adjusting the gain of the variable gain amplifier to maintain themagnitude level controlled analog signal within the input range windowof the analog-to-digital converter, the gain selected based on previewimage data acquired prior to the image; recording both the digitaloutput signals of the analog-to-digital converter and associated valuesof gain of the controllable gain amplifier in a manner to be useable torecreate the image with a range in excess of that of the window;calculating data for a tone mapped image of the at least one image, by aprocessor, from quantities representative of magnitudes of the levelcontrolled analog signal within the input range window and magnitude ofthe adjusted signals within the input range window and their associatedadjustment quantities; and outputting, by the processor, data of thetone mapped image via an output, wherein the data of the tone mappedimage is further generated based on data generated by ananalog-to-digital converter and data associated with photo exposure timeof the at least one image.
 30. The method of claim 29, whereindetermining whether the level controlled analog signal has a magnitudeoutside the window and adjusting the gain of the variable gain amplifierare both carried out for magnitudes of individual pixels of thephotosensor analog signal.
 31. The method of claim 29, whereindetermining whether the level controlled analog signal has a magnitudeoutside the window and adjusting the gain of the variable gain amplifierare both carried out for magnitudes representing individual groups ofpixels that include a plurality of adjacent pixels.
 32. The method ofclaim 29, wherein determining whether the level controlled analog signalhas a magnitude outside the window includes comparing the photosensoranalog signal with at least one analog reference level.
 33. The methodof claim 29, wherein determining whether the level controlled analogsignal has a magnitude outside the window includes comparing the outputdigital signals of the analog-to-digital converter with at least onedigital reference level.
 34. A method of processing electronic signalsof at least one optical image that has a dynamic range of magnitudesoutside of a predefined range of magnitudes, comprising: acquiringpreview image data having a reduced resolution that is less than aresolution of the at least one optical image, wherein the preview imagedata is acquired to set gain for acquiring image signals of the at leastone optical image; adjusting magnitudes of the image signals outside ofthe predefined range by quantities selected to maintain the adjustedsignals within the predefined range during acquisition of them, thequantities selected based on preview image data acquired prior to theimage signals; processing the image signals by utilizing at least oneimage processing algorithm with magnitudes of the image signals withinthe predefined range, and with quantity adjusted magnitudes of the imagesignals outside of the predefined range, and thereafter combining theresults; calculating data for a tone mapped image of the at least oneoptical image, by a processor, from the quantities representative of themagnitudes of the image signals within the predefined range and themagnitudes of the adjusted signals within the predefined range and theirassociated adjustment quantities; and outputting, by the processor, dataof the tone mapped image via an output, wherein the data of the tonemapped image is further generated based on data generated by ananalog-to-digital converter and data associated with photo exposure timeof the image signals.
 35. The method of claim 34, wherein the associatedpredefined range adjustment quantities are employed subsequent to thecombining the results.
 36. The method of claim 34, wherein theassociated predefined range adjustment quantities are employed duringthe combining the results.
 37. The method of claim 34, wherein theassociated predefined range adjustment quantities include an imagesignal gain.
 38. The method of claim 34, wherein the associatedpredefined range adjustment quantities include a quantity related to atime of exposure during acquisition of the image signals.
 39. The methodof claim 34, wherein the image processing includes use of a de-mosaicingalgorithm.
 40. The method of claim 34, wherein the image processingincludes use of a white balance algorithm.
 41. The method of claim 34,wherein the image processing includes use of a noise reductionalgorithm.
 42. The method of claim 34, wherein the image processingincludes use of a color mapping algorithm.
 43. The method of claim 34,wherein the image processing includes use of a digital lightingalgorithm.
 44. The method of claim 34, wherein the image processingincludes use of a highlight recovery algorithm.
 45. The method of claim34, wherein the image processing includes use of a tone mappingalgorithm.
 46. The method of claim 34, wherein the image processingincludes use of a JPEC compression algorithm.
 47. The method of claim34, wherein the image processing includes use of a HDR imagereconstruction algorithm.
 48. The method of claim 34, additionallycomprising reconstructing at least one image from quantitiesrepresentative of the magnitudes of the processed image signals withinthe predefined range, and the magnitudes of the processed adjustedsignals within the predefined range along with their associatedadjustment quantities.
 49. The method of claim 34, additionallycomprising storing in a non-volatile memory quantities representative ofthe magnitudes of the processed image signals within the predefinedrange, and the magnitudes of the processed adjusted signals within thepredefined range and their associated adjustment quantities.
 50. Themethod of claim 34, wherein the predefined range of magnitudes isdefined by a limitation of a system on which the signal processing takesplace.
 51. The method of claim 34, wherein the signals that representmagnitudes based upon individual pixels of the optical image and theadjustment quantities are associated with the individual pixels.
 52. Themethod of claim 34, wherein the signals that represent magnitudes basedupon individual pixels of the optical image and the adjustmentquantities are associated with blocks that individually include aplurality of individual pixels.
 53. A method of acquiring and processingdata of an optical image from a source, comprising: acquiring previewimage data having a reduced resolution that is less than a resolution ofdata for the optical image, wherein the preview image data is acquiredto set gain for acquiring the data of the optical image from the source;dividing a continuum of potential magnitudes from the source into aplurality of discrete ranges; representing magnitudes from the source bydata of values within individual ranges and associated data thatidentify the ranges in which the data of values lie, the associated dataselected based on preview image data acquired prior to the data for theoptical image from the source; individually processing the data ofvalues within the individual ranges without regard to the identity ofthe ranges in which they lie; thereafter combining the processed datafrom multiple ranges; calculating data for a tone mapped image of theoptical image, by a processor, from quantities representative of themagnitudes of data for the optical image within an individual range andthe magnitudes of the data of values within the individual range andassociated adjustment quantities; and outputting, by the processor, dataof the tone mapped image via an output, wherein the data of the tonemapped image is further generated based on data generated by ananalog-to-digital converter and data associated with photo exposure timeof the data for the optical image.
 54. The method of claim 53, whereincombining the processed data additionally includes combining the datawith the processed data with the identity of the ranges in which thedata lie.
 55. The method of claim 53, wherein the data are acquired froma source including a photosensor having an output of magnitudes thatrepresent a luminance of an optical image cast onto the photosensor. 56.The method of claim 53, wherein dividing the continuum of potentialmagnitudes from the source includes defining a plurality of discreteranges of magnitudes that are the same, the discrete ranges ofmagnitudes being selected to be within the processing capabilities of adevice that processes the data of values within the individual ranges.57. A digital camera, comprising: a photosensor having an electricaloutput proportional to a luminance of an optical image incident thereon;an optical system that casts an optical image of an object scene ontothe photosensor; an analog-to-digital converter that generates digitaldata from an analog signal within a defined window of values at an inputthereof; an adjustable gain amplifier receiving the analog signal as anoutput of the photosensor and connected with the input of ananalog-to-digital converter, the gain of the amplifier being controlledto maintain an input to the analog-to-digital converter within thewindow thereof, the gain selected based on preview image data acquiredprior to acquiring the analog signal of the optical image of the objectscene; and a processor that utilizes both the data generated by theanalog-to-digital converter and data of the gain of the amplifier togenerate data to represent the luminance of the optical image cast ontothe photosensor, wherein the processor executes instructions that enableactions, including: acquiring preview image data having a reducedresolution that is less than a resolution of the analog signal of theoptical image, wherein the preview image data is acquired to set gainfor the amplifier that is acquiring the analog signal of the opticalimage; calculating data for a tone mapped image from the quantitiesrepresentative of magnitudes of the optical image's analog signalswithin a predefined range for the window of values and the magnitudes ofthe adjusted analog signals within the predefined range and theirassociated adjustment quantities; and outputting data of the tone mappedimage via an output, wherein the data of the tone mapped image isfurther generated based on data generated by an analog-to-digitalconverter and data associated with photo exposure time of the opticalimage's analog signals.
 58. The camera of claim 57, additionallycomprising an output through which the data of the image cast onto thephotosensor are passed, including data of the gain of the amplifier. 59.The camera of claim 57, wherein the data of tone mapped images arepassed through the data output without data of the gain of theamplifier.